Distributed redundant adaptive cluster

ABSTRACT

Efficiency in the operation of a data archive and media life cycle management system is achieved by utilizing multiple system cores to avoid potential failures and appropriately distribute activities, along with multiple communication channels for efficiently moving information. The two system cores operate in a redundant system fail-over mode, thus insuring continuous operation. The two communication channels include different operating characteristics, one cable of low cost communication, while the other capable of higher cost/higher data rate communication. In the application where large amounts of data are moved between components, efficiency is achieved by having the higher cost/higher data rate network move this large amount of information, while the lower data rate/lower cost communication system allows instructions to easily be communicated, thus coordinating operations of the overall system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Provisional Application No.60/787,889 filed Mar. 31, 2006.

BACKGROUND OF THE INVENTION

The present invention generally relates to the storage and retrieval ofelectronic data in a computer managed data archive. More specifically,the present invention provides for the efficient storage and retrievalof audio and video data suitable for distribution over a broadcastingnetwork in a media lifecycle management system.

Broadcast facilities employ a wide variety of electronic equipment toreceive, process and transmit audio-visual content to audiences. A keycomponent in a broadcast content delivery system is a data storagearchive and media lifecycle management system where audio-visual contentis stored both before and after it is transmitted for broadcast. Datastorage archives can include a wide variety of storage media andassociated hardware such as magnetic tape, optical disk and hard diskdrive arrays. The various data storage devices used within abroadcasting facility are interconnected, via a computer network, tosupply devices which receive data from external sources. Examples ofthese supply devices includes production equipment which prepare andedit broadcast material and video servers which are used to play-outmaterial to broadcast transmitters. A distinguishing characteristic of adata archive and media lifecycle management system, compared with atypical computer network, is the tremendous amount of data thatconstitutes broadcast quality video when it is converted to a digitalform and the ongoing need for transporting such large volumes of dataamong various devices within the system in a timely manner.

The various storage hardware and media have different performance andcost characteristics which often determines how they are used within abroadcast content system. Video servers, which feature high datatransfers speeds and are relatively expensive, are used as play-outdevices to transfer electronic video data to broadcast transmitters. Onthe other end of the performance/cost spectrum are analog magnetic tapeswhich are inexpensive but limited in data access speed. There is asimilar trade off in performance versus cost for the data channels andassociated transmission protocols which are used to connect the variousstorage devices with each other. Also, trade-offs exist for the types ofconnections used to interconnect to the computer(s) which control theoverall broadcast content management system. At the low end of thespectrum of performance is ethernet cable used in standard computernetworks to connect network servers to client workstations which isreadily available at low cost. A typical high performance data channeltechnology is a fiber channel, which is faster but commensurately moreexpensive. Based on these options/trade-offs, it is desirable toconstruct a data archive and media lifecycle management system with anoptimal blend of performance and cost for both data storage and datatransmission hardware.

In addition to employing a cost effective architecture of storage andtransmission hardware, the media lifecycle management systems must alsobe reliable and flexible. Broadcast content typically includes bothprogram content for audiences as well as commercial advertising which isa major source of revenue for broadcasters. Any system down time willnegatively impact a broadcaster's immediate revenue as well as theirreputation with audience members. It is important therefore that themedia lifecycle management system be reliable and robust on both acomponent level as well as on an overall system level. This includes aconsideration of the future growth and maintenance of the system. Theability to make changes to the system, without interruption inoperation, is especially important at a time like today when rapidadvances are being made in computer and related digital storagetechnologies.

In a Referring now to FIG. 1, a prior art data archive and medialifecycle management system 1 is shown schematically. This systemillustrates the traditional approach to media data management that hasbeen employed to date. The system is comprised of various componentswhich cooperate to perform the tasks of storing, indexing, searching,and retrieving data as needed in day to day operations. A centralizedsystem core 2 contains software which communicates with and controlsexternal data storage devices such as a digital video and/or mediaservers (DVS) 6, a system memory cache 7, and a media library 8. Themedia library is shown having a number of separate memory modules whichrepresent an organized plurality of devices such as magnetic tapedrives, hard disk drives, optical drives, holographic drives or anyother suitable data storage devices. System core 2 is also connected toan external database 3, which contains a searchable index of the storagelocations of all data stored by the system. Typically the index recordsin the database will contain descriptive metadata related to the datastored on the various external storage components 6-8. In addition,database 3 contains an active database of the configuration of the dataarchive and media lifecycle management system 1 as well as operation andstatus logs. The system core 2 is connected to the various externalcomponents through two separate data channels. The dashed lines 9represent internet protocol (IP) infrastructure and could be eitherEthernet or GigEthernet which are capable of data transfer rates of10-100 Mbits/sec for the former and 1 Gbit/sec for the latter. The solidlines 10 represent Fiber Channel (FC) which have a significantly higherdata transfer capacity in the range of 1-4 Gbits/sec. Expected advancesin FC technology will lead to yet higher data transfer capacities of 8Gbits/sec within several years and a further increase to 16 Gbits/secseveral years after that.

Several approaches to data storage and media lifecycle issues have beendeveloped in the past. Several of these include:

U.S. Pat. No. 4,296,465 discloses a method of moving substantial blocksof data from an I/O buffer or disk cache buffer disk (external) memoryto working processor memory for use by a program running in the computersystem. The method involves the use of an autonomous hardware device, aso-called data mover, to affect the transfer of data. The data mover canreceive and store instruction words and data words and is provided withthe necessary registers, counters, switches and digital logic circuitsso that it can issue appropriate memory retrieval and storage (read andwrite) instructions. The data mover is provided with a throttle tominimize potential conflicts with other components of the dataprocessing system. It is designed so that its read and writeinstructions overlap each other, significantly increasing the rate ofdata transfer. The data mover relieves the control processors of thedata processing system of the responsibility of moving blocks of datafrom the I/O buffer to the working store thereby significantlyincreasing the rate of data transfer.

U.S. Pat. No. 5,091,849 describes a computer image production systemthat features computer workstations connected to data image storagedevices which contain image data that is retrieved and processed by theworkstations. The workstations are connected to the storage devices bytwo separate communication networks; a first network for sending andreceiving system logistics and control information to the data storagedevices and a second network for transferring data files from storage tothe workstations and vice versa.

U.S. Pat. No. 5,860,026 involves a method of transferring data betweenclusters in a system comprising a plurality of clusters by issuing datatransfer instructions from a first cluster which are carried out by adata transfer processing unit on a second cluster without having to usea main data processor in the second cluster. This allows the system totransfer data between clusters without compromising the general dataprocessing ability of clusters affected by the data transfer.

U.S. Pat. No. 5,917,723 discloses a method of mirroring write requestdata from a first device to a second device where the second device hasa primary and a secondary processor. Variable length control data isincluded in the data stream to the second device and is received by thesecondary processor which reads and executes the control data withoutrequiring a hardware interrupt of the primary processor of the seconddevice.

U.S. Pat. No. 5,944,789 involves a network file server consisting of acached disk array and a plurality of data mover computers for providingaccess to data files stored in the cached disk array. The data moversperform file system tasks such as mapping file names to logical datablocks, and the locking and unlocking of data files to minimize theloading on the cached disk array. Each data mover maintains a localcache of the file directory software including locking information oflocked files that are accessible through the data mover. Data transfersmay use more than one data mover simultaneously to expedite the transferof data from the cached disk array to a client computer. A cacheconsistency scheme ensures that file locking information is consistentamong the local caches of the plurality of data mover computers.

U.S. Pat. No. 6,324,581 describes a file server system in which aplurality of data mover computers control access to data stored onrespective data storage devices. A network client can access data storedin the system through any of the data movers. If a data mover receives arequest for data whose access is controlled by another data mover, thesecond data mover performs a lock on the requested data and sendsmetadata for the requested data to the first data mover computer whichthen uses the metadata to formulate a data access command to retrievethe requested data over a data path which bypasses the data movercomputer which controls access to the requested data.

U.S. Pat. No. 6,651,130 outlines a computer system interface between ahost computer/server and a storage device comprising an array of diskdrives. The interface includes a plurality of front end directorscoupled to the host computer server and a plurality of back enddirectors coupled to the array of disk drives. The interface alsocomprises a data transfer section containing a cache memory and amessaging network that is used to send control signals between the frontand back end directors. Each director includes a data pipe fortransmitting data to and from the cache memory in the data transfersection, a microprocessor, a controller and a common bus for connectingthe aforementioned components. The controller manages the exchange ofmessages between the front and back end directors used to control thetransfer of data through the data pipes to the cache memory in the datatransfer section.

U.S. Pat. No. 6,735,717 describes a distributed computing clusteringmodel which features at least two file servers each of which uses avirtual shared memory containing data objects or tuples which can beaccessed independently by worker processes running on client computerdevices. A state table is used to synchronize the active virtual sharedmemory of a first active file server with the stand-by virtual memory ofthe stand-by file server. In the event of a failure, the stand-by fileserver will become active and the worker processes will be able toaccess data objects or tuples in its version of the shared virtualmemory.

U.S. Pat. No. 6,779,071 discloses a computer interface between a hostcomputer/server and a storage device comprising an array of disk drives.The interface includes a plurality of front end directors coupled to thehost computer server and a plurality of back end directors coupled tothe array of disk drives. The interface also comprises two independentcommunication networks, one for transmitting control messages betweenthe front and back end directors and a second for transmitting datathrough a common cache memory between the front and back end directors.

U.S. Pat. No. 6,990,606 involves a method for effecting a failoverchange of control from one node in a loosely connected cluster of nodesin a distributed computer environment to another candidate failovernode. Configuration data for a plurality of candidate failover nodes ismaintained in a data store and status messages are sent from each of thecandidate failover nodes, analyzed and compared with the configurationinformation to determine if a failure in the current controlling nodehas occurred and if a failover change of control should be initiated.The method provides for a cascading failover which provides for a firstfailover candidate taking over only a portion of the processes of afailed control node with a second failover candidate taking over theremaining functions.

U.S. Pat. No. 7,010,575 describes a method of transferring data betweena host computer and a bank of disk drives over an interface whichcomprises two separate communication networks connecting front end datatransfer directors connected with the host computer and back end datatransfer directors connected with the bank of disk drives. The interfacemediates and controls the transfer of data from the host computer to thedisk drives and vice versa. One of the communication networks in theinterface is used to pass control messages from the front end directorsto the back end directors and vice versa while the second communicationnetwork is reserved for the transfer of data only from the host computerto the disk drives or vice versa through an intermediary cache memory.The use of a separate network for messaging relieves the cache memoryfrom the burden of storing and transmitting the control messages andimproves the operation of the system. The method includes the provisionof checking where messages received by a second director are from avalid first director, and if not, terminating the data transfertransaction.

U.S. Pat. No. 7,028,218 illustrates a redundant file server system whichuses a plurality of processor boards each having at least 2 physicalprocessors each of which share a cache memory, multiple functional unitsand a common register unit allowing each of the physical processors tobe programmed to act as separate logical processors. The separatelogical processors on each processor board are programmed to function asnetwork control processors, data storage control processors and datamover processors in either an active functioning mode or in a stand-bymode ready to take over from the active processor in the event of afailure.

BRIEF SUMMARY OF THE INVENTION

The present invention is a novel hardware and software architecture fora data archive and media lifecycle management system which achieves anumber of improvements over prior art systems. Specifically, thearchitecture and methods of the present invention provides greaterefficiency in the use of resources, more flexible bandwidth management,system robustness and scalability. These improvements are especiallyimportant in data archive systems which regularly handle very largeamounts of audio-visual data while maintaining extremely high standardsof reliability. The above-mentioned results are achieved by using adistributed data archive system architecture which has a plurality ofsystem cores operated as a redundant fail-over capable cluster. At leasttwo data communication networks are included in the system, with a firstnetwork for transmitting data movement control instructions and a secondnetwork for transmitting content data. At least one data mover isconnected to the core system cluster via the first communication networkand to a plurality of data storage devices via the second communicationnetwork. In this way the data mover will efficiently manage data storageactivities by effectively utilizing these two communication networks. Ina preferred embodiment the first communication network uses low coststandard ethernet network cabling and the second communication networkuses high bandwidth fiber optic channel.

As mentioned above, the system of the present invention utilizes aplurality of system cores to operate as a redundant fail-over capablecluster. This is achieved by having multiple cores which are incommunication with one another, and which operate in a coordinatedmanner. A first core is traditionally in control of system operations,and is capable of assigning tasks related to data management andmovement. A second system core, can handle certain assigned tasks, whilealso monitoring the operation of the first system core. Should failuresbe detected in any way, the second system core is capable of assumingcontrol of system operations. At this time, modifications andadjustments to the first system core can be achieved to correct existingproblems. Once operational, the first system core can then againmaintain system control.

In addition, the present invention efficiently utilizes data movers toachieve the above-mentioned efficiency. System cores coordinate withmultiple data movers to achieve movement of the large data files relatedto audio-visual data. By utilizing these separate systems, efficient useof the second communication network is achieved, thus speeding theability to move necessary information.

BRIEF DESCRIPTION OF THE DRAWINGS

The inventive data archive and media lifecycle management system will bedescribed below in greater detail by reference to the attached drawings

FIG. 1 is a schematic block diagram depicting the data storage archiveand media lifecycle management system architecture typical of prior artsystems

FIG. 2 is a schematic block diagram showing the system architecture of adata storage archive and media lifecycle management system in accordancewith the present invention

FIG. 3 is a schematic block diagram that depicts the distribution offunctions and processes between a system core processor and a data moverin a preferred embodiment.

FIG. 4 is a schematic block diagram which shows the material movementmanager in more detail with specific end point managers connecting todata storage devices.

FIG. 5 is a flow chart illustrating the flow of control of a data moverin the inventive data archive and media lifecycle management system.

FIG. 6 is a schematic block diagram which depicts the configuration ofthe two core systems.

FIG. 7 is a schematic diagram illustrating the fail-over process betweenredundant system cores within the inventive data archive and medialifecycle management system including Table 1 which describes systemsevents and their effects on the current state of a system core.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to FIG. 2, a data archive and media lifecycle managementsystem 21 is shown schematically which incorporates the novel featuresof the invention. To operate efficiently, a first system core 20 and asecond system core 22 are incorporated into the system, which hasconnections via IP infrastructure 9 and Fiber Channel 10 to externaldata storage locations, including digital video and/or media servers 6,system cache 7 and media library 8. Although only two system cores areillustrated, the system is not limited to two and can in factaccommodate further additional cores as needed by the requirements ofthe system. The second system core 22 is connected to and communicateswith the first system core 20 and the system database 3. The two systemcores 20 and 22 co-operate as components of an adaptive cluster toprovide system control redundancy, which greatly helps to enhance thereliability and robustness of the system. Only one of the two coresystems is in active control of the data archive and media lifecyclemanagement system 21 at any one time. The other core system continuallymonitors the active core system and if there are indications that theactive core system is not operating correctly, the non-active core willtake control of system 21 and shut down the active processes running inthe previously active core system. For example, assuming first systemcore 20 is initially active, second system core 22 will operate tomonitor activities on the system 21 until a problem is identified. Atthat time, second core system 22 will assume control. Once active,second core system 22 will attempt to recover first core system 20through the initiation of various measures including restoring power andif successful, first core system 20 will then resume the role of thenon-active, stand-by core system. The use of two or more identical coresystems, each capable of operating in either an active or stand-by mode,greatly enhances the reliability and robustness of the system. Thisconfiguration is also advantageous in that it allows service personnelto maintain and upgrade each core system, in turn, without disruptingthe operation of the data archive and media lifecycle management system.Thus the amount of system down-time from either unanticipated failure orregular maintenance is minimized by the use of multiple system cores 20,22 operating as an adaptive cluster.

FIG. 6 depicts schematically the configuration of the two core systems,20 and 22. Each system contains a clustering manager 32 and 52 whichremains active regardless of the active or non-active status of eachhost core system. The clustering manager of the active core systemcontinuously sends status data of all running processes to the externaldatabase, and the non-active core system continuously polls this data todetermine if the active core system is functioning properly. If there isan indication of failure, the non-active system core executes afail-over routine which transfers control of the archive system to it,and shuts down all application functional processes left operating onthe recently failed core system. This includes shutting down all datamovers and any data transfers in progress. This termination step isincluded in the fail-over routine to ensure that only one core system isever in control of the system and the various external storage devicesat any one time. After the termination step, the new active coreinitiates its other system modules and attempts to restart thepreviously active data transfers.

The process of fail-over discussed above is illustrated schematically inmore detail in FIG. 7 where the various potential states of a systemcore within an adaptive cluster are represented as components in adigital state machine. Table 1 accompanies FIG. 7 and describes systemsevents and their effects on the current state of a system core. Thecluster state machine represented in FIG. 7 applies to the operation ofall of the potential 1 to N system cores connected to data archive andmedia lifecycle management system 21. Again, the redundant system coresconfigured as cluster state machines ensures overall system robustnessand reliability.

In data archive and media lifecycle management system 21 the transfer ofdata from one source or storage location to another involves twoseparate steps. Initially there is a preparation step which involvessignaling from the first system core 20 to both the source anddestination of the data to be transmitted. First core system 20communicates with database 3 to determine the address(es) of the data tobe transmitted and relays this information to the data source, thencommunicates with the data destination to determine which address(es)are suitable for receiving the data. First core 20 then performsendpoint preparation to ensure that both the source and destination areready to send and receive the data. The endpoint preparation proceduresvary depending upon the type of device that is involved. For example ifthe receiving endpoint is a tape drive, the endpoint preparation stagewill involve ensuring that a tape cassette is mounted in a suitabledrive and has sufficient capacity to receive the expected data. Afterthe source and destination are thus prepared and ready to transmit datafirst system core 20 initiates the second step of streaming ortransmitting the data.

As discussed above, prior art systems perform both the signaling andstreaming of data sequentially on one type of network channel. Forexample first system core 20, depending on the status of the dataarchive and media lifecycle management system, may perform bothsignaling and transmission over the FC network, or conversely it mayperform both operations over the IP network. The disadvantage of this isthat signaling is typically a low bandwidth operation and combining itwith high bandwidth data transmission over the same type of networkchannel is inherently inefficient. In a data archive and media lifecyclemanagement system which exists to regularly store and retrieve verylarge amounts of data this inefficiency can significantly affect theoverall performance and/or cost of the system.

Returning now to FIG. 2, a set of external data movers 23 are connectedto data archive and media lifecycle management system 21 via both IP andFC networks. The data movers 23 are stand alone hardware componentswhich specialize in transmitting large volumes of data specifically overhigh capacity data channels such as Fiber Channel(s) 10. Each of thecore systems 20, 22 are enabled with software modules that perform thesignaling functions for all data transfers. After the signaling processis completed control is passed to one or more independent data movers 23for actual transmission of the data from one the source location to thedestination. The separation of the data movement process into twodistinct steps allows the signaling to be communicated over lowerbandwidth IP channel(s) while reserving the higher performance FCchannel(s) for high bandwidth data transfer. A further benefit ofseparating the signaling from the transmission of data is that the datamovers become modular units that can be added, removed, upgraded orreconfigured at will. This flexibility allows the system to be scalablein terms of overall system bandwidth. Additionally the modular nature ofthe data movers allows them to be employed in groups of more than one inany particular data transfer and thus can achieve an aggregation ofbandwidth resulting in faster data transmission for higher prioritytransfers.

FIG. 3 schematically depicts the distribution of functions and processesin a preferred embodiment of data archive and media lifecycle managementsystem 21. The system core 2 contains clustering manager module 32 aswell as a material movement manager 34 and a data mover manager 33 andis connected via IP infrastructure 9 to an external data mover 23.Although only one data mover 23 is depicted, there can be multiple datamovers 23 connected to first system core 20 as is needed by therequirements of the particular data archive and media lifecyclemanagement system 21. Within data mover 23 are a data mover processmanager 37 and data mover process 38. Data mover process manager 37 isresponsible for starting and stopping the data mover process and thedata mover operating system. The data mover manager module 33 managesthe configuration of data movers allowing system administrators theoption of adding, removing or configuring the operational parameters ofthe individual data movers. In addition, the data mover manager 33monitors the functional status of each data mover and ensures that theyare capable of operating. A further function of data mover manager 33 isto manage the bandwidth of the system by allocating the flow of datawithin the system to the most appropriate data movers. Data movermanager 33 actively monitors the flow of data and can re-route datatransfers from one data mover in response to overloading or failure ofspecific data movers. This ability to actively respond to failures ofindividual data movers makes the system an adaptive one that issignificantly more robust than prior art systems. The architecture ofthe system further allows data movers to be hot-swappable, that is, datamovers can be added or removed from the system while it is operating andengaged in data transfers without affecting the system or the processesactively running on it.

The material movement manager 34 contains a number of endpoint managers39 which communicate with and prepare the various data storage devicesand servers connected to the data archive and media lifecycle managementsystem for the transfer of data from one to another. FIG. 4 shows thematerial movement module 34 in more detail with specific end pointmanagers including: a storage subsystem endpoint manager 40 whichmanages communication and preparation of a system cache device 7; amedia library endpoint manager 41 which manages communication andpreparation of media library 8; and a digital video and/or media serverendpoint manager 42 which manages communication and preparation ofdigital video and/or media servers 6. Each endpoint manager 40, 41, and42 communicates with its respective external device via lower bandwidthIP infrastructure 9. Also shown in FIG. 4 is the connection between thematerial movement manager 34 and the data mover manager 33 and itsconnection to an external data mover 36, both connections using lowerbandwidth IP infrastructure. Data mover 23 is shown being connected toeach of the external data sources/servers by high bandwidth FCinfrastructure 10. Endpoint managers 40, 41, and 42 will collectinformation from the various storage devices such as file address, filesize etc. and create a configuration object which is passed on to datamover 23 via data mover manager 33. Once data mover 23 has configurationobjects for both the data source and destination, and the endpoints areprepared, it can perform the data transfer independently from the systemcore over high bandwidth fiber channel 10. The initiation of the datatransfer starts with a signal from the material movement manager 34 tothe data mover selected for the transfer by the data mover manager 33.

FIG. 5 is a flow chart illustrating the flow of control of a data moverin data archive and media lifecycle management system 21. A series ofprocess actions and tests are conducted upon system initializationstarting with step 100 which is an attempt to get a reference to thedata mover process manager 37 within a data mover 23. A failure to get areference will lead to a system restart otherwise the system will entera running, but inactive state where it waits for a data transferinitiation. When a data transfer is initiated, a data mover process 38is started and an attempt is made to get a process reference at step200. A failure will lead to a test of system communications at step 300followed by either a process or system restart. A successful retrievalof a data mover process reference leads to the next step of configuringendpoints 400. A failure will trigger another system communications test500 followed by a system and/or process restart or a connection problemnotification. A successful configuration of transfer endpoints leads toa test at step 600 of the endpoints and if successful leads to the startof a data transfer. During data transfer a transfer completion test 700is performed to ensure that the endpoints remain connected and a failureleads to an attempt to test and/or reconfigure all the endpointsinvolved in the transfer or to a restart of the process and/or system ifnecessary. A successful transfer completion test leaves the data moverand endpoint pair ready for the next data transfer operation.

Unlike prior art systems, the system architecture of the inventive dataarchive and media lifecycle management system 21 allows for theseparation of the data transfer process into two distinct and separateprocesses, each using appropriate hardware in the most efficient manner.Another benefit of this architecture is that it is easily scalable;additional data movers can be added to the system to increase overallsystem bandwidth if needed and the presence of a plurality of datamovers provides valuable system redundancy in the event of a data moverfailure. In the case of a data mover failure the data mover can beremoved from the system without affecting the remainder of the system orthe processes actively running on it.

Although the description of the preferred embodiment of the inventionmentions specifics with regards to the type of data communicationnetworks that are employed to transmit data movement control data andaudio video content data it is contemplated that other types of datacommunication networks could be used without deviating from the spiritof the present invention. It is intended that the scope of the presentinvention be determined by the appended claims and not by thedescription of the preferred embodiment.

1. A distributed data archive and media lifecycle management systemcomprising: a first system core and a second system core managed as anadaptive cluster with either of the first system core or the secondsystem core being active and in control of the system at any one time,the first system core and the second system core both providing datamovement control capabilities; a plurality of data storage devices usedfor storing data within the system, the plurality of data storagedevices connected to said first system core and second system core overa first data communication network; a database containing systemconfiguration data and storage location data for all data stored on theplurality of storage devices, the database connected to said firstsystem core and second system core over said first data communicationnetwork; at least one data mover connected to said first system core viasaid first data communication network and connected to said plurality ofdata storage devices over a second data communication network, each ofsaid at least one data mover configured to receive storage location dataand data transfer commands over said first data communication networkand perform data transfers from said plurality of data storage devicesover said second data communication network based on said storagelocation data and data transfer commands; wherein the first system corecommunicates with said plurality of data storage devices over said firstdata communication network preparing them for writing or reading data,and wherein said first system core also communicates with said databaseover said first data communication network obtaining data storageaddress data which is transmitted to said at least one data mover overthe first data communication network, thereby enabling said at least onedata mover to transfer data to or from the data storage devices over thesecond data communication network using the data storage address data.2. The system in claim 1 wherein the data bandwidth capacity of thesecond data communication network is greater than that of the first datacommunication network.
 3. The system in claim 2 wherein the first datacommunication network uses a low bandwidth Ethernet data channel.
 4. Thesystem in claim 2 wherein the second data communication network uses ahigh bandwidth fiber optic data channel.
 5. The system in claim 1wherein the first system core and the second system core are able tomonitor the functional status of each data mover active in the dataarchive system.
 6. The system in claim 5 wherein the first system coremonitors the flow of data within the data archive system and upon theexistence of predetermined conditions re-routes data transfers from onedata mover to another.
 7. The system in claim 6 wherein the first coresystem is capable of adding or removing data movers from the dataarchive system.
 8. The system in claim 1 wherein the plurality of systemcores each contain clustering management capability that is active atall times.
 9. The system in claim 8 wherein the clustering managementcapability of the active system core transmits process data on allrunning processes under its control to a database where it is stored andregularly polled by the active clustering management software of thenon-active system cores to determine if a failure in the operation ofthe active system core has occurred.
 10. A data archive system for thestorage and management of large amounts of data, comprising: a pluralityof system cores operating as an adaptive cluster, wherein a first systemcore is active and in control until another of the plurality of systemcores detects that the first system core is not operational, at whichtime a second system core of the plurality of system cores will becomeactive and in control until the first system core is operational; aplurality of data storage devices used for storing and archiving dataprovided to the data archive system; a storage system databasecontaining system configuration data and location data, wherein thelocation data indicates the location of data stored on the plurality ofdata storage devices; at least one data mover for coordinating thetransportation of data to and from the plurality of data storagedevices; a first communication network coordinating communicationbetween the plurality of system cores, the storage system database, andthe at least one data mover, wherein the first communication networkwill communicate transfer instructions indicating where data is to bestored and delivered; and a second communication network coordinatingcommunication between the first plurality of cores, the plurality ofstorage devices and the data mover, wherein the second communicationnetwork is configured to transport data to and from the data storagedevices based on the transfer instructions.
 11. The data archive systemof claim 10 wherein the second communication network has a databandwidth capacity which is higher than the data bandwidth capacity ofthe first communication network.
 12. The data archive system of claim 11wherein the second communication network is a fiber channel network. 13.The data archive system of claim 11 wherein the first communicationnetwork is an Ethernet network.
 14. The data archive system of claim 10wherein the data storage device is used to store audio-visual data. 15.The data archive system of claim 10 wherein the plurality of systemcores include a cluster management system in which the active systemcore transmits process data reporting all running processes to a clustermanagement database, the cluster management system regularly polling thecluster management database to determine the potential existence of asystem failure.
 16. The data archive system of claim 15 wherein thecluster management system is active at all times.
 17. The data archivesystem of claim 16 wherein the cluster management system will transfercontrol to the second core system upon the detection of a failure by thecluster management system.
 18. The data archive system of claim 10further comprising a second data mover capable of operating inconjunction with the existing data mover.
 19. A distributed data archivesystem comprising: two or more system cores providing an adaptivecluster, including a primary system core performing a plurality ofprocesses to actively manage data storage activities in the distributeddata archive system, and secondary system cores configured to monitorthe primary system core and actively manage the data storage activitiesin the distributed data archive system upon operational failure of theprimary system core; one or more data storage devices containing datastored within the distributed data archive system, the one or more datastorage devices connected to the two or more system cores via a lowerspeed data communication network topology; one or more databasescontaining storage information for data stored on the one or more datastorage devices, the one or more databases connected to the two or moreof system cores via the lower speed data communication network topology;one or more data movers used for transferring data within thedistributed data archive system, the one or more data movers connectedto the two or more system cores via the lower speed data communicationnetwork topology and connected to the one or more data storage devicesover a higher speed data communication network topology; wherein thelower speed data communication network topology is used to communicatedata storage commands and data storage information between the two ormore system cores, the one or more data storage devices, the one or moredatabases, and the one or more data movers; and wherein the higher speeddata communication network topology is used by the one or more datamovers to transfer data to and from the one or more data storage devicesbased on the data storage commands and storage information received bythe one or more data movers via the lower speed data communicationnetwork topology; thereby enabling data commands and signaling to betransmitted over the lower speed data communication network topology,and data transfers from the data storage devices to be transmitted overthe higher speed data communication network topology.